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Detailed Action 
Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1,114, including the fee set 
forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 
5/23/2006 has been entered. 

Response to Amendment 

2. The amendments filed on 5/23/2006 have been entered. The amendments 
overcome the previous grounds of rejection under 35 USC 112 second paragraph. 

Response to Argument 

3. The arguments filed on 5/23/2006 concerning 35 USC 101 on page 12 of the 
response are persuasive in view of the amendments made to claims 13, 16, and 23. 
The arguments filed on 5/23/2006 concerning 35 USC 1 12 on page 13 of the response 
are not persuasive for the reasons set forth below in the 35 USC 112 rejection. 

Claim Rejections - 35 USC §112 

4. The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 
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5. Claims 13. 16-23, 25, 26, 29-35, and 37-43 are rejected under 35 U.S.C. 112. 
second paragraph, as being incomplete for omitting essential steps, such omission 
amounting to a gap between the steps. See MPEP § 2172.01 . The omitted steps are: 
Claim 13: 

This claim fails to claim the user moves the real instrument in response to 
viewing the displayed at least one graphic path guide on a stereo display overlaid onto a 
view that contains said real instrument and a real object, which includes said target. 

Claim 16: 

This claim fails to claim the user views the rendered graphical representation of 
the actual target point and the actual path in the form of a graphical target point and at 
least one graphical axis marker on a display such that augmented reality combines a 
view of an actual scene with said graphical representation. This claim fails to claim the 
user moves the actual instrument in response to viewing the rendered graphical 
representation of the actual target point and the actual path in the form of a graphical 
target point and at least one graphical axis marker on a display along with the view of 
an actual scene. 

Claims 17-22, 42. and 43: 

These dependent claim do not add the missing essential steps to claim 16. 
Claim 19 refers to a user but does not add the missing essential step of having the user 
move the actual instrument. 
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Claim 23: 

This claim fails to claim the user moves the actual instrument in response to 
viewing the rendered graphical representation of the actual target point, virtual 
instrument and virtual target point on a display. 

Claims 25, 26, 31-35, and 37-41: 

These dependent claims do not add the missing essential steps to claim 23. 
Claims 31-34 refers to a user but does not add the missing essential step of having the 
user move the actual instrument. Claim 32 in the last line does not clearly identify the 
"selection". 

6. The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

7. Claims 13, 16-23, 25, 26, 29-35, and 37-43 are rejected under 35 U.S.C. 112, 
first paragraph, as failing to comply with the enablement requirement. The claim(s) 
contains subject matter which was not described in the specification in such a way as to 
enable one skilled in the art to which it pertains, or with which it is most nearly 
connected, to make and/or use the invention. As discussed above the claims fail to 
claim the user moves the real instrument, thus, they cover having an automated control 
means move the real instrument. The specification does not teach how to move the 
real instrument by automated control means in response to the position of the target 
and the instrument because this would require complex image analysis which has not 
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been disclosed by applicants' application. Thus, the specification, while being enabling 
for the user in response to viewing the display moving the real instrument, does not 
reasonably provide enablement for automated movement of the real instrument. The 
specification does not enable any person skilled in the art to which it pertains, or with 
which it is most nearly connected, to build the invention commensurate in scope with 
these claims. Therefore, the claim scope needs to be limited to the user moving the 
real instrument while viewing the displayed graphics and the real object. 

Double Patenting 

8. Claims 13, 16-23, 25. 26, 29-35, and 37-43 of this application conflict with claims 
1,13, and 25 of Application No. 09/818,388. 37 CFR 1.78(b) provides that when two or 
more applications filed by the same applicant contain conflicting claims, elimination of 
such claims from all but one application may be required in the absence of good and 
sufficient reason for their retention during pendency in more than one application. 
Applicant is required to either cancel the conflicting claims from all but one application 
or maintain a clear line of demarcation between the applications. See MPEP § 822. 

Claim Rejections - 35 USC § 102 

9. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
states. 
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10. Claims 13, 16-20, 23, 25, 26, 29, 30. 35, 37, 39, 41, 42, and 43 are rejected 
under 35 U.S.C. 102(b) as being anticipated by applicants admission of the prior art. 
See applicants specification at pages 1-5 and page 34 line 16 to page 36 line 8. 
Applicants admission discusses using several types of graphical markers in the 
augmented reality display to allow the user to align the actual instrument with the path 
needed to be taken to place the instrument onto the actual target. 
Claim 13: 

Applicants admission of the prior art bulls eye graphical marker teaches a 
method for augmented reality guided instrument positioning, comprising the steps of: 

rendering at least one graphics path guide for indicating a path for a real 
instrument to follow to a target {Bulls eye.); 

displaying the rendered at least one graphics path guide on a stereo display 
overlaid onto a view that contains said real instrument and a real object, which includes 
said target {display 124, page 34 line 16 to page 36 line 8); 

moving said real instrument to align it with the at least one graphics path guide 
{page 34 line 16 to page 36 line 8), 

aligning the real instrument with the path by determining when the at least one 
graphics path guide frames the path so that a view of a central part of the real 
instrument is not obstructed by the at least one graphics path guide {Bulls eye.); and 

moving said real instrument along the path so that a front portion of said 
real instrument is inserted into the object until its tip reaches said target {page 34 line 16 
to page 36 line 8). 
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Claim 16: 

Applicants admission of the prior art bulls eye graphical marker teaches a 
method for augmented reality guided instrument positioning, comprising the steps of: 

defining a point on an actual target (page 34 line 16 to page 36 line 8); 

defining an actual path to reach the point on the actual target {page 34 line 16 to 
page 36 line 8); 

rendering a graphical representation of the actual target point and the actual path 
in the form of a graphical target point and at least one graphical axis marker, 
respectively, the graphical representation being rendered with respect to a user's 
augmented reality viewpoint, wherein the augmented reality view combines a view of an 
actual scene with said graphical representation and the augmented reality line of sight 
to the graphical target point coincides with the actual path that the actual instrument 
needs to be aligned to during a positioning of the actual instrument to the actual target 
point {page 34 line 16 to page 36 line 8); and 

aligning the actual instrument to the actual path by aligning it to the augmented 
reality line of sight towards the graphical target point, and moving the actual instrument 
along the actual path towards the actual target point while keeping it aligned with the 
augmented reality line of sight {page 34 line 16 to page 36 line 8). 

Claim 17: 

Inherently the actual instrument includes at least one physical axis marker for 
alignment with said at least one graphical axis marker. 
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Claim 18: 

Inherently the actual instrument includes at least one physical axis marker for 
alignment with said at least one graphical axis marker. 
Claim 19: 

Page 34 line 16 to page 36 line 8 teaches a user makes the augmented reality 
line of sight coincide with the actual path by moving into a position where said graphical 
target point and said at least one graphical axis marker line up. 

Claim 20: 

Page 34 line 16 to page 36 line 8 teaches the at least one identified graphical 
axis marker has a circular shape, and is centered on the axis of the actual path. 

Claim 23: 

Applicants admission of the prior art bulls eye graphical marker teaches a 
method for virtual reality guided instrument positioning, comprising the steps of: 

defining a point on an actual target {page 34 line 16 to page 36 line 8); 

defining an actual path to reach the point on the actual target (page 34 line 16 to 
page 36 line 8); 

tracking a pose of an actual instrument with respect to a pose of the actual target 
{page 34 line 16 to page 36 line 8); 

rendering a graphical representation of the actual instrument and the actual 
target point to obtain a virtual instrument and a virtual target point, respectively, the 
graphical representation being rendered with respect to a virtual viewpoint from which a 
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virtual line of sight to the virtual target point coincides with a virtual path for the virtual 
instrument to follow during a positioning of the actual instrument to the point on the 
actual target, the virtual path corresponding to the actual path, the virtual instrument 
comprising a 3D structure for line of sight alignment, the 3D structure comprising a 
plurality of markers centered on and distributed along an axis of the virtual instrument 
{page 34 line 16 to page 36 line 8); 

displaying the rendered virtual instrument and virtual target point {display 124, 
page 34 line 16 to page 36 line 8); 

aligning the virtual instrument along the virtual line of sight to the virtual target 
point in order to accordingly align the actual instrument along the actual path {page 34 
line 16 to page 36 line 8); and 

moving the actual instrument along the actual path towards the actual target 
keeping the correct alignment by observing and keeping the alignment of virtual 
instrument and virtual path {page 34 line 16 to page 36 line 8). 

Claim 25: 

Page 34 line 16 to page 36 line 8 teaches the virtual target point has a circular 
shape (a center of a target implies a circular target, page 35 line 7.). 
Claim 26: 

Page 34 line 16 to page 36 line 8 teaches wherein the circular shape is a ring. 
Claim 29: 

Page 34 line 16 to page 36 line 8 teaches said plurality of markers comprise at 
least two rings, centered on an axis of the virtual instrument. 
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Claim 30: 

The bulls eye of page 34 line 16 to page 36 line 8 teaches the two rings have 
different diameters. 
Claim 35: 

Page 34 line 16 to page 36 line 8 teaches wherein the rendering step further 
comjDrises the step of rendering graphical information about a distance between the 
actual instrument and the point on the actual target, the graphical information about the 
distance being overlaid onto the graphical representation. 

Claim 37: 

Page 34 line 16 to page 36 line 8 teaches wherein the virtual target point and the 
virtual instrument are designed such that information corresponding to the distance 
between the actual instrument and the point on the actual target can be directly 
observed from an alignment of the virtual target point and the virtual instrument. 

Claim 39: 

Page 34 line 16 to page 36 line 8 teaches wherein the virtual target point and the 
virtual instrument are each comprised of at least one ring centered on the target point 
respectively on the axis of the instrument, and a diameter of the at least one ring is 
dimensioned to achieve a pre-defined configuration together with the actual instrument 
when the actual instrument reaches the actual target. 

Claim 41: 

Page 34 line 16 to page 36 line 8 teaches wherein said graphical representation 
from the virtual viewpoint is combined with an augmented reality view. 
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Claim 42: 

Page 34 line 16 to page 36 line 8 teaches wherein the graphical target point and 
the graphical axis marker are designed such that information corresponding to the 
distance between the actual instrument and the point on the actual target can be directly 
observed from an alignment of the graphical target point and the graphical axis marker. 

Claim 43: 

Page 34 line 16 to page 36 line 8 teaches wherein the graphical target point and 
the graphical axis marker are each comprised of at least one ring centered on the target 
point respectively on the axis of the instrument, and a diameter of the at least one ring is 
dimensioned to achieve a pre-defined configuration together withy the actual instrument 
when the actual instrument reaches the actual target. 

11. Claims 16, 22, 23, 31-35. 37, and 40-42 are rejected under 35 U.S.C. 102(b) as 
being anticipated by Kienzle, III. US PGPub application no. 2002/ 0077540. 
Claim 16: 

Kienzel teaches a method for augmented reality guided instrument positioning, 
comprising the steps of: 

defining a point on an actual target (figure 7, 159); 

defining an actual path to reach the point on the actual target (156); 

rendering a graphical representation of the actual target point (159) and the 
actual path (156) in the form of a graphical target point and at least one graphical axis 
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marker (156 or 131 or 21 1), respectively, the graphical representation being rendered 
with respect to a user's augmented reality viewpoint, wherein the augmented reality 
view combines a view of an actual scene with said graphical representation and the 
augmented reality line of sight to the graphical target point coincides with the actual 
path that the actual instrument needs to be aligned to during a positioning of the actual 
instrument to the actual target point (figures 4-7, 1 1 , and 12); and 

aligning the actual instrument to the actual path by aligning it to the augmented 
reality line of sight towards the graphical target point, and moving the actual instrument 
along the actual path towards the actual target point while keeping it aligned with the 
augmented reality line of sight {The user moves the instrument in response to viewing 
the display which shows the alignment of the instrument with the required path,). 

Claim 22: 

Kienzel teaches the method according to claim 16, wherein the at least one 
identified graphical axis marker comprises at least two axis markers for controlling 
alignment of the actual instrument along a line of sight (see figures 4-7, 1 1 , and 12 ). 

Claim 23: 

This claim is similar to claim 16 and it is rejected for the same reasons. 
Claim 31: 

Kienzel teaches the method of claim 23, wherein the step of aligning the virtual 
instrument further comprises the step of choosing an orientation of the graphical 
representation around the virtual line of sight according to a pose of a user with respect 
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to the actual target {by sensing the orientation of the real instrument the system has 
sensed the pose of the user). 
Claim 32: 

Kienzel teaches the method of claim 31 , further comprising the step of 
determining the orientation such that east, west, north, and south correspond to right, 
left, forward, and backward, respectively, for the pose of the user in which the user 
faces the actual target, said determining step based on a selection {by sensing the 
orientation of the real instrument the system has sensed the pose of the user). 

Claim 33: 

Kienzel teaches the method of claim 31, wherein the orientation is dynamically 
adjusted according to a change of the pose of the user {by sensing the orientation of the 
real instrument the system has sensed the pose of the user). 

Claim 34: 

Kienzel teaches the method of claim 32, wherein the selection is dynamically 
adjusted with respect to the pose of the user {by sensing the orientation of the real 
instrument the system has sensed the pose of the user). 

Claim 35: 

Kienzel teaches the method of claim 23 wherein the rendering step further 
comprises the step of rendering graphical information about a distance between the 
actual instrument and the point on the actual target, the graphical information about the 
distance being overlaid onto the graphical representation by graphically displaying the 
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instrument as it is moved by the user relative to the target which shows the distance 
graphically. 

Claim 37: 

Kienzel teaches the method of claim 23 wherein the virtual target point and the 
virtual instrument are designed such that information corresponding to the distance 
between the actual instrument and the point on the actual target can be directly 
obsen/ed from an alignment of the virtual target point and the virtual instrument by 
graphically displaying the instrument as it is moved by the user relative to the target 
which shows the distance graphically. 

Claim 40: 

Kienzel teaches the method of claim 23, wherein the graphical representation 
from the virtual viewpoint is combined with another graphical representation from 
another virtual viewpoint looking at the virtual path from a side thereof with reference to 
figures 4-7, 11, and 12. 

Claim 41: 

Kienzel teaches the method of claim 23. wherein said graphical representation 
from the virtual viewpoint is combined with an augmented reality view with reference to 
figures 4-7, 11, and 12. 

Claim 42: 

Kienzel teaches the method of claim 16 wherein the graphical target point and 
the graphical axis marker are designed such that information corresponding to the 
distance between the actual instrument and the point on the actual target can be directly 
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observed from an alignment of the graphical target point and the graphical axis marker 
by graphically displaying the instrument as it is moved by the user relative to the target 
which shows the distance graphically. 



Claim Rejections - 35 USC § 103 

12. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

13. Claim 21 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kienzle, 
III. US PGPub application no. 2002/ 0077540 and applicants admission of the prior art 
and motivation of using aiming aids of guns with instrument guidance found at page 38 
lines 7-16. Kienzel does not teach the graphical axis marker comprises an intersection 
of at least two lines, the intersection to be centered on the axis of the actual instrument 
for correct alignment. 

It would have been obvious to one of ordinary skill in the art to use the feature of 
gun aiming, the cross, and apply it to augmented reality aiming of real instruments 
because this will give the user a good idea of the alignment of the real instruments with 
the real target. 
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14. Claim 38 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kienzle, 
III. US PGPub application no. 2002/ 0077540. Kienzel does not teach rendering by 
using a virtual camera with a wide angle lens. Page 50 line 24 to page 51 line 4 of 
applicants specification does not give a reason for rendering by using a virtual camera 
with a wide angle lens. 

It would have been obvious to one of ordinary skill in the art to render by using a 
virtual camera with a wide angle lens because this is one of many ways to render the 
graphical images and because applicant did not give any reason for choosing this type 
of rendering. 

Conclusion 

15. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. Kienzle, III. US PGPub application no. 2005/ 01 19561 . 

1 6. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffery A Brier whose telephone number is (571) 272- 
7656. The examiner can normally be reached on M-F from 7:00 to 3:30. If attempts to 
reach the examiner by telephone are unsuccessful, the examiner's supervisor, Michael 
Razavi. can be reached at (571 ) 272-7664. The fax phone Number for the organization 
where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
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Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 




Jeffery A Brier 
Primary Examiner 
Division 2628 



